Method for automatically identifying at least one user of an eye tracking device and eye tracking device

ABSTRACT

The invention relates to a method for automatically identifying at least one user of an eye tracking device ( 10 ) by means of the eye tracking device ( 10 ), wherein user identification data of the at least one user are captured by a capturing device ( 14 ) of the eye tracking device ( 10 ). Under the first condition that at least one profile (P 1 , P 2 , P 3 ) with associated identification data (I 1 , I 2 , I 3 ) of at least one specific user is stored in a storage medium ( 20 ), the stored identification data (I 1 , I 2 , I 3 ) of the at least one profile (P 1 , P 2 , P 3 ) are compared with the captured user identification data, and under the second condition that the captured user identification data match the stored identification data (I 1 , I 2 , I 3 ) according to a predefined criterion, the at least one user is identified as the at least one specific user, for whom the at least one profile (P 1 , P 2 , P 3 ) was stored.

The invention relates to a method for automatically identifying at leastone user of an eye tracking device by means of the eye tracking deviceand a corresponding eye tracking device.

Methods for user identification are known from the prior art like fromMicrosoft's Kinect for the Xbox. Here the user identification isconsidered as a key component of Microsoft's Kinect for the Xbox, torecognize and track player identity. It can learn a player's appearanceover time in a game session. The identification can be associated with asign in profile in the Xbox system.

User identification for such devices usually is used to determine,whether users are authorized for using that devices or not, or to loaduser specific settings like most recently played games or favouritegames and so on.

Also eye tracking devices and eye tracking systems are known from theprior art. Eye tracking devices are configured to track persons,especially their eyes, their point of regard or gaze direction. Also, inthis connection profiles may be stored for specific users, however fordifferent purposes as those described above. User calibration isessential to all mainstream eye tracking systems, which can achieve 0.5to 1 degree accuracy in gaze angle. A calibration process would normallyrequire the user to look at several preselected points on a screen, andthen a set of user specific parameters can be calibrated and stored forthe further eye tracking in a profile. As certain properties of humaneyes are different from person to person, usually the calibration resultof one person will not produce accurate eye tracking for another one.Therefore the current eye tracking systems usually generate a profilefor each user to store the calibration result and some other specificsetting. This allows the user to load his/her calibration data from theprevious session instead of doing a new calibration.

Known eye tracking devices suffer from a number of drawbacks due to thefact that, before a user actually can use the eye trackingfunctionalities of his eye tracking system he has to perform severaltasks. In practice, a user has to register a new profile before doing acalibration. Furthermore, then the calibration procedure has to beperformed, which normally takes about 15 seconds to 30 seconds, whichvaries from system to system, and also depends on the experience andability of the user. When the same user comes to use the system later,he or she has to select the profile manually so that the system willload the stored calibration data. If a wrong profile is selected by theuser, the wrong calibration data will severely degrade the trackingaccuracy and the error could be more than 5 degree in gaze angle. Sincethe user has to tell the system, which profile to use, it can be a realburden and dramatically degrade the user experience when, for example,the users change frequently or the user is unable to use mouse/keyboardeasily.

Moreover, the classical user calibration procedure used by themainstream eye tracking systems is rather interruptive and unfriendlyfor certain application contact. It is believed that such classicalcalibration is one of the major barriers preventing the eye trackingtechniques to be integrated into more applications.

Therefore it is an object of the present invention to facilitate the useof eye tracking devices and reduce the efforts a user has to make forusing an eye tracking device.

This object is solved by a method for automatically identifying at leastone user of an eye tracking device by means of the eye tracking devicewith the features according to claim 1 and an eye tracking device forautomatically identifying at least one user with the features of claim15. Advantageous embodiments of the invention are presented in thesubclaims.

According to the invention the method for automatically identifying atleast one user of an eye tracking device by means of the eye trackingdevice comprises the steps:

-   a) Capturing user identification data of the at least one user by a    capturing device of the eye tracking device;-   b) Under the first condition that at least one profile with    associated identification data of at least one specific user is    stored in a storage medium, comparing the stored identification data    of the at least one profile with the captured user identification    data; and-   c) Under the second condition that the captured user identification    data match the stored identification data according to a predefined    criterion, identifying the at least one user as the at least one    specific user, for whom the at least one profile was stored.

Therefore advantageously users can be automatically identified by theeye tracking device without having to choose their respective profilesmanually. This reduces the overall effort for using an eye trackingdevice enormously. The capturing device of the eye tracking device canfor example comprise an imaging device, like a camera or an imagesensor, by means of which the user identification data, like certainuser characteristics, can be easily captured so that the eye trackingdevice does not even need additional components to perform thisautomatic user identification. Furthermore, by means of this automaticuser identification also eye tracking relevant data stored in thecorresponding profiles can be automatically derived from this profileafter the user has been identified and used for subsequent eye tracking.So if, for example, a calibration has already been performed for a user,then any time later this user can just sit in front of his eye trackingdevice and make use of it without any further tasks like choosingmanually his corresponding profile. This is especially advantageous ifmultiple users use the same eye tracking device. Different users canchange frequently without having to choose their respective settingsevery time. Also it is not even necessary to register a new profilemanually before doing a calibration as by capturing user identificationdata, these data can also be stored in a profile automatically and beused any time later. Another great advantage of this automatic useridentification, especially in connection with the use of the eyetracking device, is that it can be avoided that a user accidentallychooses the wrong profile and then wrong calibration data are used foreye tracking.

The user identification data can comprise, for example, image dataand/or numerical values of features of the at least one user. Especiallyimage data can be easily captured by the eye tracking device andanalysed to derive user characteristics therefrom which allow theidentification of the user.

Especially, the captured user identification data of the at least oneuser and/or the identification data of the specific user comprise atleast one of:

-   -   a face image and/or a face characteristic;    -   a body image and/or a body characteristic, especially at least        one geometric property of the body;    -   eye properties, especially at least one geometric property of at        least one eye of the user;    -   an eye motion pattern;    -   a voice characteristic; and    -   information about corrective glasses.

All of these characteristics can advantageously be used for identifyinga user. Most of them even can be derived from captured images so thatthe eye tracking device advantageously does not need any additionalcomponents. However, for further enhancing the reliability of correctuser identification additional components can be used as well, like amicrophone for capturing a voice characteristic of the user.

According to one advantageous embodiment of the invention, thepredefined criterion in step c) consists of a feature matching methodfor the identification data, and a threshold. In this embodiment, allelements of identification data will be converted into numerical featuredata, including the image features computed from the face images and thenumerical values describing the eye properties. To compare two sets ofsuch numerical features, a similarity score can be computed. A thresholdis then selected so that a higher score indicates that the data incomparison are collected from the same person. When there are multiplematching candidates from different profiles, the system can choose thebest match or let the user to decide.

According to another advantageous embodiment of the invention, thepredefined criterion in step c) can be based on a speech recognitionmodule. The system in this embodiment asks the user to speak a code wordto identify him or her to the system when a profile is created. Theidentification data is the audio recording of the user's voice. Thesystem will consider that the current user matches to the storedidentification data, if the speech recognition module detects the codeword pattern in the captured audio.

According to another advantageous embodiment of the invention in step a)for capturing the user identification data at least one image of atleast one body part of the at least one user is captured, especially bymeans of an imaging device of the capturing device, and the useridentification data are determined on the basis of the captured image,especially by means of the processing unit of the eye tracking device.Deriving user characteristics from captured images is especiallyadvantageous as already explained, because no additional components ofthe eye tracking device are required. The eye tracking device simply cancapture images of the user, by means of the camera for example, and theprocessing unit processes these images to derive the user identificationdata therefrom. Moreover, eye tracking devices, and preferably also theeye tracking device according to this invention, usually have theability to capture images of the whole face of large part of the face ofa user anyway. Therefore, especially face recognition techniques can beapplied very easily to derive the user identification data form theimage.

According to another advantageous embodiment of the invention the atleast one profile comprises profile data including the associatedidentification data of the specific user and user specific control datafor the specific user. The stored identification data are used, asexplained above, to be able to re-identify the user, for who these datawere stored. However, the great advantage of this embodiment is, thatalso additional data, which are not used for identifying a user, can bestored in the profile, like calibration data, which can then also beautomatically be derived from the profile and used by the eye trackingdevice after the user has been identified.

According to another advantageous embodiment of the invention forstoring the at least one profile of the specific user a calibrationprocedure is performed, during which at least one calibration parameterof the specific user is determined and stored as the control data orpart of the control data in the user profile for the specific user. Inparticular that calibration parameter can contain a gaze correctioninformation for the specific user, like an offset between a visual axisand an optical axis of the at least one eye of the specific user. Duringsuch a calibration procedure not only the calibration parameters can bedetermined and stored, but also the user identification data can bedetermined in such a calibration procedure at the same time as well andstored in the corresponding profile. So a user does not even have toregister his profile itself, as everything can be done automatically bythe eye tracking device during a calibration procedure. So if a new useruses the eye tracking device and performs a calibration procedure theeye tracking device automatically stores a profile for this user,derives the identification data from e.g. the images captured during thecalibration procedure, derives the calibration data or calibrationparameters necessary for eye tracking from this calibration procedure,and stores everything in the corresponding profile. This user can thenuse the eye tracking device any time later without having to perform anytasks, as the eye tracking device automatically can identify the useragain on the basis of the stored identification data, and automaticallyderive the calibration parameters from his profile and the user canimmediately use the eye tracking functionalities of the eye trackingdevice.

According to another advantageous embodiment of the invention, if instep c) the at least one user is identified as the specific user, theprofile data, especially the at least one calibration parameter, is usedfor eye tracking of the at least one user. This calibration parametercan for example be used to calculate the gaze direction of the user orthe point of regard, for example on a screen of the eye tracking device.Due to the automatic user identification it can be avoided that a userchooses accidentally the wrong profile, so that especially when derivingthe calibration parameters from the respective automatic determinedprofile detrimental impact on the eye tracking accuracy by using wrongcalibration data can be avoided.

According to another advantageous embodiment of the invention thecalibration procedure is performed as an implicit calibration procedure,especially wherein an actual point of regard of the at least onespecific user is estimated on the basis of image content of images shownon a display device of the eye tracking device. This is an especiallyadvantageous embodiment as in this case the user does not even noticethat a calibration procedure is performed at the moment. In a classicalcalibration procedure usually points are shown one after the other atdifferent positions on a screen and the user is asked to fixate thesepoints. During the user is fixating each of these points the eyetracking device calculates the point of regard of the user and comparesthe calculated points of regard with the position of the shown points onthe screen, which is assumed to be the actual point of regard. Such aprocedure can be very boring for a user, is very time consuming andrequires complete concentration of the user. Instead in the implicitcalibration procedure no points are shown, which the user has to fixate.In such an implicit calibration procedure the eye tracking devicecalculates the point of regard of a user and estimates the actual pointof regard, namely the point on the screen the user is actually lookingat the time the eye tracking device calculates the point of regard, onthe basis of image content shown on the screen of a display device atthat moment. If the user, for example, works on a computer and moves themouse cursor, it can be assumed that also the eyes of the user willfollow that mouse cursor, so that the position of the mouse cursor canbe estimated as the actual point of regard of the user. If the user isfor example watching a film, it can be assumed that the eyes of the userwill look on faces or eyes of persons shown in the film, or on themouths of persons speaking in that film. Such an implicit calibrationusually takes more time, but has the great advantage, that a user isactually not aware of the fact that at the moment a calibrationprocedure is performed.

These features of performing an implicit calibration in connection withthe previously described embodiments, especially the automatic useridentification, the automatic storing of profiles with identificationdata and calibration data have the great advantage that a user of an eyetracking device now does not have to perform any tasks anymore formaking use of the eye tracking functionalities. He does not have toregister profiles, to choose his profile, to actively perform acalibration, to choose his profile again if in between another user hasused the eye tracking device, and so on.

Moreover, the control data, like the calibration parameters, which arestored in the profile besides the identification data, can comprisefurther very advantageous data, which can be used for eye tracking orfor using the eye tracking device. Therefore it is an especiallyadvantageous embodiment of the invention when the control data relate toat least one of the following.

-   -   the at least one calibration parameter of the specific user;    -   an information about corrective glasses of the specific user,        especially if the specific user wears corrective glasses or not        and/or which refractive power the corrective glasses comprise;    -   rights and/or permissions of the specific user, especially that        define the range of abilities and/or functions a user identified        as a specific user is allowed to perform on a system comprising        the eye tracking device;    -   a priority of the specific user, especially with respect to        another specific user, wherein the priority of the specific user        with respect to the other specific user defines, the interaction        of which of the specific users' interaction with the system has        priority, especially in case of contradicting interactions.

As with regard to the information about corrective glasses, which can,as explained above also be used as identification data, this informationcan also be used as control data, for example by processing unit of theeye tracking device. For example, the refractive power of correctiveglasses can influence the calculated gaze direction, so knowing thisrefractive power can be used to correct the calculated gaze directioncorrespondingly to improve the eye tracking accuracy. To have rights,permissions or priorities of specific user saved in the profile isespecially advantageous with regard to multiple users using the same eyetracking device. So on the basis of these data the eye tracking devicecan decide which user is allowed to do which tasks and which users havepriority over others.

According to another advantageous embodiment of the invention under thethird condition that the at least one user is identified as the specificuser in step c) a processing unit of the eye tracking device iscontrolled in dependency of the control data or the profile of thespecific user, the at least one user was identified as. This controllingcan be done as described above. If the user is identified, theprocessing unit can read out the control data of the corresponding userprofile and control the eye tracking device or any other system coupledwith eye tracking device in dependency of these data, for examplecalculating the gaze direction or a point of regard taking into accountthe stored calibration parameters, information about corrective glasses,allowing or prohibiting certain tasks for the identified user accordingto his rights or priority data, and so on. Especially the processingunit controls at least one system parameter of the eye tracking deviceor a system, which comprises the eye tracking device, in dependency ofthe permissions and/or rights and/or priority of the specific user, theat least one user is identified with.

According to another advantageous embodiment of the invention, if noprofile is stored of if the captured user identification data do notmatch the stored identification data of the at least one profile, a newprofile, especially with the captured identification data, for the atleast one user is stored. So if no profile is stored or the captureduser identification data do not match, the eye tracking devicerecognizes the user as a new user and can automatically store a newprofile for this user with the corresponding captured useridentification data. Moreover, in these cases, the eye tracking devicealso can automatically initiate the calibration procedure, preferably asan implicit calibration procedure, for the at least one user. So,advantageously new user profiles can be created automatically withoutrequiring any active action of the user.

According to another advantageous embodiment of the invention if the atleast one profile is the only profile stored in the storage medium, theprocessing unit determines for checking whether the second condition isfulfilled, whether the identification data of the profile match thecaptured user identification data of the at least one user within apredetermined threshold, and if several profiles with associatedidentification data are stored, the processing unit for checking whetherthe second condition is fulfilled performs a feature matching bydetermining that profile from the stored profiles, the identificationdata of which have the smallest deviation from the captured useridentification data of the at least one user, in particular whereinafter performing the feature matching the processing unit checks whetherthe identification data of the determined profile match the captureduser identification data of the at least one user within a predeterminedthreshold.

In other words, if only one profile is stored and the eye trackingdevice detects a user, the eye tracking device only has to determinewhether the captured user identification data match the storedidentification data sufficiently, i.e. within that predeterminedthreshold. If several profiles are stored, the processing unit can firstperform, for example a nearest neighbour matching, to choose the profilewhich fits best. After having found the best profile candidate it can beverified whether the identification data of this profile match the useridentification data sufficiently, namely again by means of apredetermined threshold. This procedure works advantageously with anyarbitrary number of users and profiles. However, as the accuracy of userrecognition may drop when the number of candidates, namely differentusers, grows, it might be advantageous to restrict the number ofstorable profiles to a predefined maximum number.

Alternatively or additionally, the accuracy of user recognition can alsobe enhanced by the following advantageous embodiments of the invention:For example, if the at least one user is identified as the specific userin step c) a request for confirmation of the identification is outputtedon a display of the eye tracking device. Thereby advantageously apossibility is provided for the user to actively check whether thecorrect profile has been chosen. Also according to another advantageousembodiment of the invention, if the eye tracking device receives a userinput indicating that the identification is wrong, the eye trackingdevice stores a new profile for the at least one user for the captureduser identification data. The user input advantageously can also be usedfor an automatical learning process of the eye tracking device. Forexample, if the eye tracking device receives a user input indicatingthat the identification is wrong, the processing unit can modify atleast one parameter relating to the at least one predefined criterion inthe second condition, which is the matching condition, in a predefinedway. E.g. the processing unit can modify the above named predeterminedthreshold or parameters of the feature matching or make use of this userinteraction to improve the recognition accuracy of the eye trackingdevice in any different way.

The invention further relates to an eye tracking device forautomatically identifying at least one user, wherein the eye trackingdevice comprises a storage medium configured to store at least oneprofile with associated identification data of a specific user.Furthermore, the eye tracking device comprises a capturing device, forexample an imaging device like one or more cameras or image sensors,configured to capture user identification data of the at least one user,a processing unit configured, under the first condition that the atleast one profile with associated identification data of at least onespecific user is stored in the storage medium, to compare the storedidentification data of the at least one profile with the capturedidentification data of the at least one user, wherein the processingunit is further configured to identify the at least one user as aspecific user, for whom the at least one profile is stored, under thesecond condition that the user identification data match the storedidentification data according to a predefined criterion.

The preferred embodiments and advantages thereof described with regardto the method according to the invention correspondingly apply to theeye tracking device according to the invention, wherein in particularthe embodiments of the method constitute further preferred embodimentsof the eye tracking device. Furthermore, the invention may also relateto a system comprising the eye tracking device according to theinvention.

Further features of the invention and advantages thereof derive from theclaims, the figures, and the description of the figures. All featuresand feature combinations previously mentioned in the description as wellas the features and feature combinations mentioned further along in thedescription of the figures and/or shown solely in the figures are notonly usable in the combination indicated in each case, but also indifferent combinations or on their own.

The invention is now explained in more detail with reference toindividual preferred embodiments and with reference to the attacheddrawing. These show in:

FIG. 1 a schematic illustration of an eye tracking device forautomatically identifying users according to an embodiment of theinvention;

FIG. 2 a schematic illustration of a method for automaticallyidentifying users by means of an eye tracking device according to anembodiment of the invention;

FIG. 3 a schematic illustration of a method for automaticallyidentifying users by an eye tracking device according to anotherembodiment of the invention;

FIG. 4 a schematic illustration of a display device of an eye trackingdevice for providing feedback according to an embodiment of theinvention; and

FIG. 5 a schematic illustration of a method for automaticallyidentifying users by means of the eye tracking device in combinationwith implicit calibration according to an embodiment of the invention.

In the following, advantageous embodiments of the invention arepresented, which apply user identification techniques to improve theusability of eye tracking systems.

FIG. 1 shows a schematic illustration of an eye tracking device 10 forproviding automatic user identification according to an embodiment ofthe invention. The eye tracking device 10 comprises a processing unit12, a capturing device 14 that can comprise one or more cameras 14 a,and optional light sources 16. This eye tracking device 10 can work asknown eye tracking devices. The camera 14 a is configured to captureimages of a user, these images are passed to the processing unit 12 andprocessed, wherein certain eye features can be identified in the imagesand on the basis of these features the gaze direction and/or point ofregard of the user can be determined. The optional light sources 16 canprovide illumination, for example to produce corneal reflections on theeyes of the user, which can be detected in the images and used tocalculate the gaze direction. The eye tracking device 10 can alsocomprise a display device 18 and the processing unit 12 can beconfigured to calculate a point of regard of the user on the screen ofthe display device 18. Therefore control functionalities can be providedon the basis of the calculated gaze direction and/or point of regard. Soa user can, for example, control applications running on the eyetracking device 10 or on a system, like a computer, coupled to the eyetracking device 10 by means of his gaze similar as to using a mouse orother input devices. Moreover, the processing unit 12 comprises a userrecognition module 12 a and a GUI (Graphical User Interface) module 12b. Also, the eye tracking device 10 can comprise a storage device 20.However, this storage device 20 needs not to be part of the eye trackingdevice 10, but can also be part of another system, server, computer orthe like, with which the eye tracking device 10 can be connected. Thestorage device 20 can for example also be an internet server, like thecloud, in which data can be stored.

The processing unit 12 now is configured to store user profiles P1, P2,P3 in this storage device 20. Here exemplarily three user profiles P1,P2, P3 for three different users are shown. Each user profile P1, P2, P3contains identification data I1, I2, I3 and control data C1, C2, C3.Such identification data I1, I2, I3 can be for example facecharacteristics, body characteristics, eye characteristics, and so on,on the basis of which the different users can be identified. The controldata C1, C2, C3, like calibration parameters, permissions, rights orpriorities, can be used by the processing unit 12 for controllingpurposes, for example to calculate the gaze direction and taking intoconsideration the respective calibration data of the respective user.The recognition module 12 a matches the current user to one of theregistered profiles P1, P2, P3, if a matching condition is fulfilled.The GUI module 12 b can provide feedback to the user and allows the userfor example to correct wrong identification results. For each user, allrelevant data are stored within the corresponding profile P1, P2, P3.All the registered profiles P1, P2, P3 form a large database as thecandidate pool of user identification. The identification data I1, I2,I3 preferably constitute data, which can be derived from capturedimages, however, such identification data can also be acquiredotherwise, for example as voice characteristics captured by a microphone14 b, which can also be part of the capturing device 14 of the eyetracking device 10.

FIG. 2 shows a schematic illustration of a method for automaticallyidentifying users according to an embodiment of the invention. In thiscase it is assumed, that a user is present, for example in a capturingarea of the eye tracking device 10, which can be noticed by the eyetracking device 10 itself or e.g. by receiving a corresponding userinput or the like. So the method starts in step S10, in which the eyetracking device 10 captures an image of the user by means of the camera14 a. This image is then passed to the processing unit 12, which in stepS12 processes the image and derives user identification data from thisimage. After that, the processing unit 12 checks in step S14 whether apredefined matching condition is fulfilled. For doing so, the processingunit 12 compares the captured user identification data with theidentification data I1, I2, I3 of each of the stored profiles P1, P2,P3. If the processing unit 12 determines, that the captured useridentification data match one of the identification data I1, I2, I3according to a predefined criterion the processing unit 12 derives instep S16 the corresponding calibration data, like calibrationparameters, and so on, which are stored as part of the control data C1,C2, C3 of the profile P1, P2, P3 for which the matching condition wasfulfilled. The processing unit 12 can then use these user specificcalibration data for eye tracking. If, however, in step S14 the matchingcondition is not fulfilled for any of the stored profiles P1, P2, P3,the user is recognized as a new user and the eye tracking device 10 mayautomatically start a calibration procedure in step S18. Thiscalibration can be an explicit or implicit calibration. During thiscalibration procedure the camera 14 a captures images of the user instep S20, passes these images again to the processing unit 12 and againthe processing unit 12 processes these images in step S22. Whenprocessing these images the processing unit 12 derives useridentification data and calibration data, which are stored in step S24in a corresponding user profile P1, P2, P3. These calibration data canthen be used again for eye tracking purposes in step S16.

As an alternative, when the processing unit 12 determines, that thematching condition has not been fulfilled in step S14, the useridentification data determined in step S12 can be directly saved into acorresponding user profile P1, P2, P3 in Step S24. In parallel thecalibration can be performed, in which then only calibration data needto be determined, but not necessarily additional user identificationdata.

Moreover, this method can optionally begin with a step S0, in which theprocessing unit 12 checks whether at least one profile P1, P2, P3 isstored in the storage device 20. If no profile is stored at all, it isnot necessary to check whether a user has a corresponding profile P1,P2, P3 or not and the eye tracking device 10 can initialize acalibration procedure in step S18 right away.

According to this method advantageously a new user can initialize aprofile for the first time he or she uses the eye tracking device 10.Calibration data and image data for identification can be stored withinthis profile P1, P2, P3. Furthermore, it is possible to use this profileP1, P2, P3 on another eye tracking system, if it can be distributed.Also, when a user comes, the eye tracking device 10 tries to identifythe user. If he or she is recognized as a registered user, theassociated profile P1, P2, P3 will be loaded and the user can start eyetracking without any manual interaction. This quick response willgreatly improve the user experience, especially in the scenario thatdifferent users change frequently in using the eye tracking device 10,but preferably restricted to a small group of people. Also, if the eyetracking device 10 recognizes a user as a new user, the eye trackingdevice 10 may ask the user to initialize a new profile and do acalibration, for example by supplying feedback to the user by means ofthe display device 18.

FIG. 3 shows schematically another embodiment of a method for automaticuser identification by means of an eye tracking device.

The method shown in FIG. 3 again starts with capturing images in stepS30. These images can then be used by the processing unit 12 on the onehand for performing eye tracking in step S32 as well as for facerecognition and eye feature comparison in step S34. Especially, for facerecognition and eye feature comparison results of the eye trackingprocess in step S32 can be used.

The imaging devices in eye tracking systems usually have the ability tocapture the image of the whole face or large parts of the face, whennecessary. Therefore, many face recognition techniques can be adapted tobe used in this place, too. In addition to face recognition, thissolution can be extended to use or include other information about theuser for better recognition accuracy. The information may include thebody images of the users captured by the imaging devices, the eyeproperties of the user from the eye tracking itself, eye motion patternanalyzed from the eye tracking results, the voices of the users, and soon. Since the accuracy of the user recognition drops when the number ofcandidates grows, the system or an administrator may have to maintainthe size of the candidate pool.

These results and/or other user characteristics derived from the imagescan be compared to identification data derived from the registeredprofiles P1, P2, P3. From these registers profiles P1, P2, P3 thatprofile is selected in step S36, which matches best the captured useridentification data. Optionally this selected profile can be confirmedby user feedback FB. Afterwards, this selected profile also containscalibration parameters for the recognized user, which in step S38 areapplied to the eye data, which were outputted in step S40 as a result ofthe eye tracking process. Therefrom gaze data, the final gaze,especially gaze direction or the point of regard of the user can bedetermined in step S42.

In a more specific embodiment of this described method in step S30 thefull face image of the user can be captured, which may involve facedetection and camera control mechanism. For face recognition in step S34a histogram of oriented gradient features can be computed from the facearea after image normalization and the obtained features can be furtherprocessed with a dimension reduction technique to obtain compact anddiscriminative features. The core eye tracking method, executed in stepS32, also processes the captured images and measures several eyeproperties, e.g. iris size and iris contrast. The results are thenprovided for the recognition process executed in step S34, in which theprovided eye features are combined with the face image featuresobtained, as explained before. All users in the candidate pool havetheir corresponding feature vectors stored and their correspondingprofiles P1, P2, P3. A feature matching technique, e.g. nearest neighbormatching, can select the best candidate in step S36. A similaritymeasure between the features from the current user and the ones from thebest candidate has to be checked against a threshold to give the finalrecognition result. If the similarity is high enough, the current userhas the same identity as the best candidate. If the similarity is low,the current user is considered as a new user. This can initiate anautomatic storing of a new profile and/or a calibration procedure.

FIG. 4 shows a schematic illustration of a display screen of a displaydevice 18 of the eye tracking device 10 for providing user feedback. Theuser can be provided with the option to overwrite a wrong identificationresult of the software. For this purpose, a pop-up window 24 can beshown, e.g. in a screen corner, on the display device 18, which presentsthe identification result, e.g. by a registered photo 22, and a pair of“buttons” as choosing options to receive a feedback from the user. Herea “correct” button C and a “wrong” button W are presented. By choosingthe “correct” button C the user can confirm that the correct profile P1,P2, P3 was chosen, and by choosing the “wrong” button W the user canindicate that the wrong profile was chosen and the identificationfailed. If the user claims that the identification is wrong, then anoption can be provided to enter or select the correct identity,especially to choose the correct profile P1, P2, P3. When there is nomouse/keyboard available as input device, the user can use the gazeinteraction to give feedback, in particular by looking at thecorresponding “button”. However, without a correct calibration the eyetracking is much less accurate and therefore would require a larger areaof the display 18 to implement such an interaction mechanism.

To improve the accuracy of user identification, the face image and eyefeatures of the current user can be collected and merged into theexisting data in the profile P1, P2, P3. So, each time a user uses theeye tracking device 10 again the eye tracking device 10 can store thecaptured identification data each time for the corresponding profile P1,P2, P3. To reduce the risk of error, this should only be done when theuser feedback is available.

This way, when a user starts to use the system, i.e. the eye trackingdevice 10, with a computer display 18, the pop-up window 24 will notifythe user about his/her recognized identity. The system then loads thestored calibration data while the user can immediately start using eyetracking. The users are not forced to give feedback, although it isrecommended to do so when a wrong identity is assigned. If the userconsiders the identification result as wrong, the system can be notifiedand the user may enter his correct identity. The system, i.e. the eyetracking device 10, can also make use of this user interaction toimprove its recognition accuracy. For example, the matching parameters,like the mentioned threshold, can be modified, if the user indicates,that the identification was wrong. Automatic user identification makesthe association of the current user to an existing data collection(profile) much easier and more user-friendly. Combining this with atechnique to predict user point of regard, e.g. by analyzing the imagecontent on the display 18, one can completely hide the calibration.

The state of the art computer vision algorithm can predict the user gazeby analyzing what is shown on the display 18. A simpler case would beshowing a single moving object on a simple background. For more naturalimages or videos, more sophisticated algorithms are required.Application tailored solution and simple images or videos could lead tomore reliable results. Naturally the software cannot be sure whether theuser is looking at a predicted position, e.g. a moving object, socertain “matching” analysis has to be done to get the reliablecalibration data. Such a method may take longer time to achieve certainaccuracy. However, since the data are collected with more variantconditions, the overall performance could be even better.

An embodiment of a method for automatically identifying users incombination with such an implicit calibration is shown for a gameinvolving moving targets in FIG. 5. Here in step S50 the eye trackingdevice, especially the camera 14 a, captures images of the user andprovides these user images for user identification in step S52 and foreye tracking in step S54. In parallel, the application running on theeye tracking device 10, in this case a video game, shows image or videocontent on the display device 18 in step S56. This content is providedfor the prediction of the point of regard of the user in step S58. Ifoptionally input devices like a mouse or controller are used by a userin step S60 these data like the position of a cursor shown on thedisplay device 18, can be provided for the user point of regardprediction in step S58 as well. So while the user is playing the videogame, thereby watching the image content provided by the application,and is optionally using input devices, in the eye tracking process instep S54 the point of regard of the user is determined and compared instep S62 with corresponding predicted points of regard provided in stepS58. For this comparison a data mining process can be performed forreliable calibration data. The user identification data provided in stepS52 as well as the result of the data mining in step S62 can be storedin a corresponding user profile P1, P2, P3. This implicit calibrationprocedure can be performed until enough calibration data are captured.If then finally in step S64 it is determined that enough calibrationdata are captured, eye tracking device 10 is ready for eye tracking andthen can perform eye tracking of the user in step S66.

The classical user calibration procedure used by the mainstream eyetracking systems is rather interruptive and unfriendly for certainapplication context. This presented implicit calibration therefore is avery advantageous alternative. When an application is using thisfeature, the gaze data and the predicted point of regard are fed into apattern analysis module. The module collect calibration data, which canthen be used for the current session and also for the future usage. Theuser may start the application without the gaze/eye tracking feature orless accurate gaze/eye tracking. After using the system for a while, theaccurate gaze/eye tracking will be available.

The invention and its embodiments facilitate to improve the userexperience of the eye tracking device by adding user identificationfunction into the eye tracking device. It allows for a quick response tothe recognized users and is essential to the usability of the eyetracking device in certain multiuser scenarios. It also opens the doorto an eye tracking device using completely implicit calibration. So bythis invention great improvements with regard to user experienceregarding user data storage, especially the calibration data, can beachieved, multiuser eye tracking can be supported and the apparentnessof the calibration process can be reduced.

Special advantages can be also achieved with regard to simultaneousmultiuser eye tracking. When the eye tracking device has to trackmultiple users at the same time, the eye tracking device needs to knowwhich calibration data should be applied to which person, especiallywhich eyes of which person. This may involve user identification anduser tracking, so that especially in this case the automatic useridentification provided by the invention is very advantageous.

Another family multiuser scenario would be, for example, a boy playingan eye tracking based game and found something interesting, so he callshis father to try it, too. When the father comes, he can try the thingout immediately as long as he has used the system before. Withoutautomatic user identification, he would have to first pause the game andnotify the system to use his calibration data. Combining the automaticuser identification and the image based gaze prediction the user can becalibrated without any direct interaction. This also allows the eyetracking technique to be integrated into an existing device/applicationmuch more implicitly. The user may enjoy the cool functions brought bythe eye tracking before even noticing its existence.

With user identification, the system, namely the eye tracking device,has its own prediction for profile selection so that a registered usercan start eye tracking immediately after he or she entered the trackingrange (camera view). This will greatly improve the user experience,especially for family users, old people and people with disability.

LIST OF REFERENCE SIGNS

-   10 eye tracking device-   12 processing unit-   12 a user recognition module-   12 b GUI module-   14 capturing device-   14 a camera-   14 b microphone-   16 light source-   18 display device-   20 storage device-   22 photo-   24 pop-up-window-   P1, P2, P3 profile-   I1, I2, I3 identification data-   C1, C2, C3 control data-   C correct button-   W wrong button-   FB feedback

The invention claimed is:
 1. A method comprising: capturing a plurality of images of an eye of a user; determining, based on at least one of the plurality of images, one or more properties of the eye of the user; in accordance with a determination that a predefined matching condition is not satisfied, generating, based on the one or more properties, a user profile including eye tracking calibration data that includes gaze correction information, wherein generating the user profile includes performing a calibration procedure in order to obtain the eye tracking calibration data; and in accordance with a determination that the predefined matching condition is satisfied, determining, based on the gaze correction information, a gaze direction of the user.
 2. The method of claim 1, wherein the one or more properties includes a geometric property of the eye of the user.
 3. The method of claim 1, wherein the one or more properties includes a size of an iris of the eye of the user or a contrast of the iris of the eye of the user.
 4. The method of claim 1, wherein the gaze correction information is indicative of an offset between a visual axis of the eye and an optical axis of the eye.
 5. The method of claim 1, wherein the gaze correction information is indicative of a refractive power of a corrective lens.
 6. The method of claim 1, wherein generating the user profile includes selecting, based on comparing the one or more properties to respective properties of one or more user profiles, the user profile from the one or more user profiles.
 7. The method of claim 6, wherein comparing the one or more properties to the respective properties of the one or more user profiles includes generating respective similarity scores indicative of a similarity between the one or more properties and the respective properties of respective ones of the one of more user profiles.
 8. The method of claim 7, wherein generating the user profile includes: determining that the respective similarity score of the user profile exceeds a threshold; and retrieving the user profile from a storage medium.
 9. The method of claim 7, wherein generating the user profile includes: determining that no respective similarity score exceeds a threshold; and generating the user profile based on the one or more properties.
 10. An apparatus comprising: a camera to capture a plurality of images of an eye of a user; a processor to: determine, based on at least one of the plurality of images, one or more properties of the eye of the user; in accordance with a determination that a predefined matching condition is not satisfied, generate, based on the one or more properties, a user profile including eye tracking calibration data that includes gaze correction information, wherein generation of the user profile includes performing a calibration procedure in order to obtain the eye tracking calibration data; and in accordance with a determination that the predefined matching condition is satisfied, determine, based on the gaze correction information, a gaze direction of the user.
 11. The apparatus of claim 10, wherein the one or more properties include a geometric property of the eye of the user.
 12. The apparatus of claim 10, wherein the gaze correction information is indicative of an offset between a visual axis of the eye and an optical axis of the eye.
 13. The apparatus of claim 10, wherein the gaze correction information is indicative of a refractive power of a corrective lens.
 14. The apparatus of claim 10, wherein the processor is configured to generate the user profile by: generating respective similarity scores indicative of a similarity between the one or more properties and respective properties of respective ones of one or more user profiles; determining that the respective similarity score of the user profile exceeds a threshold; and retrieving the user profile from a storage medium.
 15. The apparatus of claim 10, wherein the processor is configured to generate the user profile by: generating respective similarity scores indicative of a similarity between the one or more properties and respective properties of respective ones of one or more user profiles; determining that no similarity score exceeds a threshold; and generating the user profile based on the one or more properties.
 16. A non-transitory computer-readable medium encoding instructions which, when executed, cause a processor to perform operations comprising: determining, based on at least one of a plurality of captured images, one or more properties of the eye of the user; in accordance with a determination that a predefined matching condition is not satisfied, generating, based on the one or more properties, a user profile including eye tracking calibration data that includes gaze correction information, wherein generating the user profile includes performing a calibration procedure in order to obtain the eye tracking calibration data; and in accordance with a determination that the predefined matching condition is satisfied, determining, based on the gaze correction information, a gaze direction of the user.
 17. The non-transitory computer-readable medium of claim 16, wherein the gaze correction information is indicative of an offset between a visual axis of the eye and an optical axis of the eye.
 18. The non-transitory computer-readable medium of claim 16, wherein the gaze correction information is indicative of a refractive power of a corrective lens.
 19. The method of claim 1, wherein the user profile includes control data, and wherein the gaze direction is determined at least in part based on the control data.
 20. The apparatus of claim 10, wherein the user profile includes control data, and wherein the gaze direction is determined at least in part based on the control data. 